watch: fix stale watcher state across log rotation - #6
Merged
Conversation
The recheck added in v1.4.14 arms the watch first and inspects the file afterwards, so a rotation can land on either side of the arming. Both sides left the tailer with a watcher it must not trust: * ChangeEvents seeded the watcher's size baseline with the caller's offset in the old file. When the name already pointed at the new, usually smaller file, the first write there looked like a truncation and the spurious reopen re-delivered every line already read from the new file. The baseline and the recorded inode now come from a stat of the file at arm time; changes that predate the watch are the recheck's job to find. * The pendingReopen and Truncated paths kept tail.changes across a reopen. When the rename fired right after the arming, the kept subscription was already dead and carried a latched Deleted notification, which triggered a bogus second reopen re-delivering everything read in between. Every reopen path now drops the subscription and re-arms it on the next EOF, and the recheck picks up whatever happened in the gap. Dropping is safe now that FileChanges.Stop shuts the producer goroutine down synchronously; an abandoned live producer would keep stealing events from the shared per-file channel. * A producer whose events channel was closed by an external RemoveWatch (e.g. Cleanup) exited silently and removed the watch a second time, driving the shared refcount negative: the tailer blocked forever, and every later watch of the same name skipped the fsnotify subscription and never got a single event. The producer now reports the file as deleted instead, and the tracker refuses to drive a refcount below zero. A stale pendingReopen is also cleared when a pending delete performs the reopen, so it cannot cause one more reopen cycle later. The new delayBeforeRecheck test knob mirrors delayBeforeWatch and widens the arm-to-recheck window for the regression tests. Part of TNTP-3131
sssciel
approved these changes
Aug 6, 2026
bigbes
added a commit
to tarantool/tt
that referenced
this pull request
Aug 7, 2026
go-tail v1.4.14 could deliver a line twice right after a log rotation: a watch armed while the file was being replaced kept a stale size baseline, so the first write to the new file looked like a truncation and forced a reopen from offset zero; a watch kept across the reopen paths could also be dead with a latched delete notification, forcing one more reopen. A watcher goroutine that lost its events channel exited silently, corrupting the shared watch refcount and stalling the tailer forever. The pinned revision is the head of tarantool/go-tail#6; switch it to the v1.4.15 tag once the fix is released. Part of TNTP-3131
Mockird31
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two
test_log_rotatefailure modes seen in tt CI after the v1.4.14 bump (first master run, 2/42 parametrizations,delay_time=0.01):fw.Size = pos). The first write to the new, smaller file then looked like a truncation, and the spurious reopen re-read the file from offset zero, re-delivering the lines already sent. A second path to the same symptom: thependingReopen/Truncatedbranches kepttail.changesacrossreopen(); when the rename fired right after the arming, the kept subscription was already dead with a latchedDeletednotification, which triggered a bogus second reopen.The fix:
ChangeEventsseeds the size baseline (and the recorded inode) from a stat of the file at arm time. In-window changes are the recheck's job (that is what it is for); the watcher only needs a baseline consistent with the file it actually watches.pendingDelete,pendingReopen,Truncated) drops the subscription via the new synchronousFileChanges.Stop()and re-arms it on the next EOF. The producer goroutine is stopped with a handshake before re-arming — an abandoned live producer would keep consuming from the shared per-file events channel, stealing events from the fresh subscription (this is why a baretail.changes = nilwould not be a safe fix).removeWatchrefuses to drive the refcount below zero.pendingReopenis cleared when a pending delete performs the reopen, so a stale flag cannot cause one more reopen cycle later.Regression tests (all reproduce on v1.4.14 on Linux, deterministically, via the existing
delayBeforeWatchknob and the newdelayBeforeRecheckknob; both inotify and polling watchers where applicable):TestReplaceWhileArmingThenAppend,TestTruncateWhileArmingThenAppend— stale baseline → spurious truncation → duplication;TestReplaceWhileRecheckingThenAppend— dead watch kept acrosspendingReopen→ double reopen → duplication;TestExternalWatchRemovalDoesNotHang— silent producer death → permanent hang;TestRotationStressExactlyOnce— 20 rotations × 5 lines at 5 ms cadence, exactly-once delivery.Verified: full suite with
-raceon Linux (docker, arm64) and macOS; new tests looped ×50 (stress) and ×8 (deterministic) plus ×10 under-racewith zero flakes.Part of TNTP-3131